Collecting h2o
  Downloading https://files.pythonhosted.org/packages/8c/6a/0ee4e2387e31cb472a952a44cffd72f1b471c51f28f0ceeeaa5e4709be1b/h2o-3.32.1.4.tar.gz (164.8MB)
     |████████████████████████████████| 164.8MB 23kB/s 
Requirement already satisfied: requests in /usr/local/lib/python3.7/dist-packages (from h2o) (2.23.0)
Requirement already satisfied: tabulate in /usr/local/lib/python3.7/dist-packages (from h2o) (0.8.9)
Requirement already satisfied: future in /usr/local/lib/python3.7/dist-packages (from h2o) (0.16.0)
Collecting colorama>=0.3.8
  Downloading https://files.pythonhosted.org/packages/44/98/5b86278fbbf250d239ae0ecb724f8572af1c91f4a11edf4d36a206189440/colorama-0.4.4-py2.py3-none-any.whl
Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests->h2o) (3.0.4)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests->h2o) (1.24.3)
Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests->h2o) (2.10)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests->h2o) (2021.5.30)
Building wheels for collected packages: h2o
  Building wheel for h2o (setup.py) ... done
  Created wheel for h2o: filename=h2o-3.32.1.4-py2.py3-none-any.whl size=164871668 sha256=920eb5f8e48b48312350aa1e12eac67b87089f7ccbba6c3a8d898841aea6fb30
  Stored in directory: /root/.cache/pip/wheels/67/76/54/89d7711536d5bb0d010bf35a34deca3eddc757efba78dbc74f
Successfully built h2o
Installing collected packages: colorama, h2o
Successfully installed colorama-0.4.4 h2o-3.32.1.4
Checking whether there is an H2O instance running at http://localhost:54321 ..... not found.
Attempting to start a local H2O server...
  Java Version: openjdk version "11.0.11" 2021-04-20; OpenJDK Runtime Environment (build 11.0.11+9-Ubuntu-0ubuntu2.18.04); OpenJDK 64-Bit Server VM (build 11.0.11+9-Ubuntu-0ubuntu2.18.04, mixed mode, sharing)
  Starting server from /usr/local/lib/python3.7/dist-packages/h2o/backend/bin/h2o.jar
  Ice root: /tmp/tmptv739hf5
  JVM stdout: /tmp/tmptv739hf5/h2o_unknownUser_started_from_python.out
  JVM stderr: /tmp/tmptv739hf5/h2o_unknownUser_started_from_python.err
  Server is running at http://127.0.0.1:54321
Connecting to H2O server at http://127.0.0.1:54321 ... successful.
H2O_cluster_uptime: 03 secs
H2O_cluster_timezone: Etc/UTC
H2O_data_parsing_timezone: UTC
H2O_cluster_version: 3.32.1.4
H2O_cluster_version_age: 10 days
H2O_cluster_name: H2O_from_python_unknownUser_7zikra
H2O_cluster_total_nodes: 1
H2O_cluster_free_memory: 3.172 Gb
H2O_cluster_total_cores: 2
H2O_cluster_allowed_cores: 2
H2O_cluster_status: accepting new members, healthy
H2O_connection_url: http://127.0.0.1:54321
H2O_connection_proxy: {"http": null, "https": null}
H2O_internal_security: False
H2O_API_Extensions: Amazon S3, XGBoost, Algos, AutoML, Core V3, TargetEncoder, Core V4
Python_version: 3.7.11 final
Parse progress: |█████████████████████████████████████████████████████████| 100%
AutoML progress: |████████████████████████████████████████████████████████| 100%

Leaderboard

Leaderboard shows models with their metrics. When provided with H2OAutoML object, the leaderboard shows 5-fold cross-validated metrics by default (depending on the H2OAutoML settings), otherwise it shows metrics computed on the frame. At most 20 models are shown by default.
model_id auc logloss aucpr mean_per_class_error rmse mse training_time_ms predict_time_per_row_msalgo
GBM_grid__1_AutoML_20210719_091245_model_1 0.79592 0.2544770.219521 0.2739970.2717190.073831 272 0.043584GBM
GBM_grid__1_AutoML_20210719_091245_model_2 0.786609 0.2885060.183858 0.2975280.2871530.0824567 296 0.042066GBM
XGBoost_grid__1_AutoML_20210719_091245_model_6 0.78225 0.2503490.211155 0.2492810.2667560.071159 141 0.028961XGBoost
XGBoost_grid__1_AutoML_20210719_091245_model_2 0.781446 0.2582670.230984 0.2821650.2711740.0735355 208 0.028001XGBoost
XGBoost_grid__1_AutoML_20210719_091245_model_5 0.771542 0.2568910.197349 0.2698070.2702260.0730219 156 0.021606XGBoost
StackedEnsemble_BestOfFamily_AutoML_20210719_0912450.770103 0.2550450.241046 0.2985020.2689170.0723163 479 0.14578 StackedEnsemble
XGBoost_grid__1_AutoML_20210719_091245_model_1 0.76858 0.328 0.193323 0.2957080.2954340.0872812 203 0.018091XGBoost
XGBoost_grid__1_AutoML_20210719_091245_model_4 0.766887 0.3108730.1851 0.3096750.2890330.0835403 161 0.017711XGBoost
XGBoost_grid__1_AutoML_20210719_091245_model_8 0.765913 0.2592830.172262 0.3411630.2693540.0725518 174 0.014107XGBoost
GBM_5_AutoML_20210719_091245 0.756264 0.2549620.188046 0.2712040.2671830.0713867 195 0.019505GBM
GBM_grid__1_AutoML_20210719_091245_model_3 0.753682 0.2554420.183356 0.2698070.2667750.0711691 186 0.02196 GBM
GBM_1_AutoML_20210719_091245 0.752497 0.2722260.192882 0.31086 0.2774570.0769822 196 0.018151GBM
XGBoost_grid__1_AutoML_20210719_091245_model_7 0.748773 0.3072670.178202 0.2698070.2919930.0852597 189 0.016829XGBoost
XGBoost_grid__1_AutoML_20210719_091245_model_9 0.748138 0.2715880.174836 0.3112830.2727890.0744139 84 0.01434 XGBoost
GBM_2_AutoML_20210719_091245 0.741366 0.2624850.196379 0.29592 0.2699720.072885 246 0.016641GBM
XGBoost_3_AutoML_20210719_091245 0.741324 0.2753590.172233 0.3327830.28101 0.0789664 129 0.015364XGBoost
GBM_grid__1_AutoML_20210719_091245_model_4 0.736711 0.2633090.178584 0.3290160.2704050.073119 131 0.016817GBM
XGBoost_2_AutoML_20210719_091245 0.736499 0.2664150.167425 0.3100980.2714490.0736844 173 0.014511XGBoost
StackedEnsemble_AllModels_AutoML_20210719_091245 0.735568 0.2689940.155055 0.3068820.2738290.0749822 365 0.141358StackedEnsemble
GBM_3_AutoML_20210719_091245 0.734256 0.2685170.170837 0.2835620.2754540.075875 168 0.026255GBM

Confusion Matrix

Confusion matrix shows a predicted class vs an actual class.

GBM_grid__1_AutoML_20210719_091245_model_1

Confusion Matrix (Act/Pred) for max f1 @ threshold = 0.2557828965438706: 
false true Error Rate
0 false 351.0 7.0 0.0196 (7.0/358.0)
1 true 8.0 25.0 0.2424 (8.0/33.0)
2 Total 359.0 32.0 0.0384 (15.0/391.0)

Variable Importance

The variable importance plot shows the relative importance of the most important variables in the model.

Variable Importance Heatmap

Variable importance heatmap shows variable importance across multiple models. Some models in H2O return variable importance for one-hot (binary indicator) encoded versions of categorical columns (e.g. Deep Learning, XGBoost). In order for the variable importance of categorical columns to be compared across all model types we compute a summarization of the the variable importance across all one-hot encoded features and return a single variable importance for the original categorical feature. By default, the models and variables are ordered by their similarity.

Model Correlation

This plot shows the correlation between the predictions of the models. For classification, frequency of identical predictions is used. By default, models are ordered by their similarity (as computed by hierarchical clustering). Interpretable models, such as GAM, GLM, and RuleFit are highlighted using red colored text.

SHAP Summary

SHAP summary plot shows the contribution of the features for each instance (row of data). The sum of the feature contributions and the bias term is equal to the raw prediction of the model, i.e., prediction before applying inverse link function.

Partial Dependence Plots

Partial dependence plot (PDP) gives a graphical depiction of the marginal effect of a variable on the response. The effect of a variable is measured in change in the mean response. PDP assumes independence between the feature for which is the PDP computed and the rest.





Confusion Matrix

Confusion matrix shows a predicted class vs an actual class.

GBM_grid__1_AutoML_20210719_091245_model_1

Confusion Matrix (Act/Pred) for max f1 @ threshold = 0.2557828965438706: 
false true Error Rate
0 false 351.0 7.0 0.0196 (7.0/358.0)
1 true 8.0 25.0 0.2424 (8.0/33.0)
2 Total 359.0 32.0 0.0384 (15.0/391.0)

Variable Importance

The variable importance plot shows the relative importance of the most important variables in the model.

SHAP Summary

SHAP summary plot shows the contribution of the features for each instance (row of data). The sum of the feature contributions and the bias term is equal to the raw prediction of the model, i.e., prediction before applying inverse link function.

Partial Dependence Plots

Partial dependence plot (PDP) gives a graphical depiction of the marginal effect of a variable on the response. The effect of a variable is measured in change in the mean response. PDP assumes independence between the feature for which is the PDP computed and the rest.





Checking whether there is an H2O instance running at http://localhost:54321 . connected.
H2O_cluster_uptime: 1 hour 36 mins
H2O_cluster_timezone: Etc/UTC
H2O_data_parsing_timezone: UTC
H2O_cluster_version: 3.32.1.4
H2O_cluster_version_age: 10 days
H2O_cluster_name: H2O_from_python_unknownUser_7zikra
H2O_cluster_total_nodes: 1
H2O_cluster_free_memory: 3.025 Gb
H2O_cluster_total_cores: 2
H2O_cluster_allowed_cores: 2
H2O_cluster_status: locked, healthy
H2O_connection_url: http://localhost:54321
H2O_connection_proxy: {"http": null, "https": null}
H2O_internal_security: False
H2O_API_Extensions: Amazon S3, XGBoost, Algos, AutoML, Core V3, TargetEncoder, Core V4
Python_version: 3.7.11 final
Parse progress: |█████████████████████████████████████████████████████████| 100%
AutoML progress: |████████████████████████████████████████████████████████| 100%

Leaderboard

Leaderboard shows models with their metrics. When provided with H2OAutoML object, the leaderboard shows 5-fold cross-validated metrics by default (depending on the H2OAutoML settings), otherwise it shows metrics computed on the frame. At most 20 models are shown by default.
model_id mean_residual_deviance rmse mse mae rmsle training_time_ms predict_time_per_row_msalgo
StackedEnsemble_AllModels_AutoML_20210719_104903 17648.9132.849 17648.9 88.62080.163532 262 0.100085StackedEnsemble
StackedEnsemble_BestOfFamily_AutoML_20210719_104903 18040.2134.314 18040.2 90.75860.164731 170 0.038168StackedEnsemble
GBM_grid__1_AutoML_20210719_104903_model_1 26298.8162.169 26298.8112.076 0.175313 348 0.030053GBM
XGBoost_grid__1_AutoML_20210719_104903_model_4 26312.2162.21 26312.2114.551 0.153739 734 0.008871XGBoost
XGBoost_grid__1_AutoML_20210719_104903_model_3 26845.6163.846 26845.6116.373 0.144884 560 0.006384XGBoost
GBM_3_AutoML_20210719_104903 28199.2167.926 28199.2108.219 0.180605 329 0.017913GBM
XGBoost_grid__1_AutoML_20210719_104903_model_5 33015.2181.701 33015.2129.204 0.125457 182 0.005574XGBoost
GBM_1_AutoML_20210719_104903 39761.3199.402 39761.3141.463 0.174056 324 0.012657GBM
XGBoost_grid__1_AutoML_20210719_104903_model_2 40280.3200.7 40280.3144.697 0.155139 427 0.007858XGBoost
XGBoost_3_AutoML_20210719_104903 40660.6201.645 40660.6145.186 0.17298 516 0.004632XGBoost
XGBoost_1_AutoML_20210719_104903 41158.9202.876 41158.9145.477 0.151495 423 0.005749XGBoost
XGBoost_grid__1_AutoML_20210719_104903_model_1 45457.6213.208 45457.6148.708 0.18636 831 0.008415XGBoost
XGBoost_grid__1_AutoML_20210719_104903_model_6 47667.2218.328 47667.2156.676 0.182955 132 0.007236XGBoost
GBM_4_AutoML_20210719_104903 49454.1222.383 49454.1148.902 0.204376 239 0.012723GBM
XGBoost_2_AutoML_20210719_104903 51269.3226.427 51269.3160.919 0.195577 148 0.005407XGBoost
GBM_grid__1_AutoML_20210719_104903_model_2 61496.6247.985 61496.6178.095 0.190502 580 0.025635GBM
GBM_2_AutoML_20210719_104903 80347.5283.456 80347.5195.946 0.22421 172 0.008235GBM
DRF_1_AutoML_20210719_104903 80365.7283.488 80365.7181.736 0.219054 342 0.006979DRF
XRT_1_AutoML_20210719_104903 90245.7300.409 90245.7196.584 0.22948 237 0.004242DRF
DeepLearning_1_AutoML_20210719_104903 135147 367.623135147 275.214 0.221364 132 0.004246DeepLearning

Residual Analysis

Residual Analysis plots the fitted values vs residuals on a test dataset. Ideally, residuals should be randomly distributed. Patterns in this plot can indicate potential problems with the model selection, e.g., using simpler model than necessary, not accounting for heteroscedasticity, autocorrelation, etc. Note that if you see "striped" lines of residuals, that is an artifact of having an integer valued (vs a real valued) response variable.

Variable Importance

The variable importance plot shows the relative importance of the most important variables in the model.

Variable Importance Heatmap

Variable importance heatmap shows variable importance across multiple models. Some models in H2O return variable importance for one-hot (binary indicator) encoded versions of categorical columns (e.g. Deep Learning, XGBoost). In order for the variable importance of categorical columns to be compared across all model types we compute a summarization of the the variable importance across all one-hot encoded features and return a single variable importance for the original categorical feature. By default, the models and variables are ordered by their similarity.

Model Correlation

This plot shows the correlation between the predictions of the models. For classification, frequency of identical predictions is used. By default, models are ordered by their similarity (as computed by hierarchical clustering). Interpretable models, such as GAM, GLM, and RuleFit are highlighted using red colored text.

SHAP Summary

SHAP summary plot shows the contribution of the features for each instance (row of data). The sum of the feature contributions and the bias term is equal to the raw prediction of the model, i.e., prediction before applying inverse link function.

Partial Dependence Plots

Partial dependence plot (PDP) gives a graphical depiction of the marginal effect of a variable on the response. The effect of a variable is measured in change in the mean response. PDP assumes independence between the feature for which is the PDP computed and the rest.





Individual Conditional Expectation

An Individual Conditional Expectation (ICE) plot gives a graphical depiction of the marginal effect of a variable on the response. ICE plots are similar to partial dependence plots (PDP); PDP shows the average effect of a feature while ICE plot shows the effect for a single instance. This function will plot the effect for each decile. In contrast to the PDP, ICE plots can provide more insight, especially when there is stronger feature interaction.





Residual Analysis

Residual Analysis plots the fitted values vs residuals on a test dataset. Ideally, residuals should be randomly distributed. Patterns in this plot can indicate potential problems with the model selection, e.g., using simpler model than necessary, not accounting for heteroscedasticity, autocorrelation, etc. Note that if you see "striped" lines of residuals, that is an artifact of having an integer valued (vs a real valued) response variable.

Partial Dependence Plots

Partial dependence plot (PDP) gives a graphical depiction of the marginal effect of a variable on the response. The effect of a variable is measured in change in the mean response. PDP assumes independence between the feature for which is the PDP computed and the rest.















Individual Conditional Expectation

An Individual Conditional Expectation (ICE) plot gives a graphical depiction of the marginal effect of a variable on the response. ICE plots are similar to partial dependence plots (PDP); PDP shows the average effect of a feature while ICE plot shows the effect for a single instance. This function will plot the effect for each decile. In contrast to the PDP, ICE plots can provide more insight, especially when there is stronger feature interaction.